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LONG-TERM  GOALS 

Our  long  term  goal  is  to  better  understand  ensemble  forecasting  in  general  and  the  dynamics  of  initial 
uncertainty  given  only  imperfect  models  in  particular;  we  are  most  interested  in  the  practical 
application  to  a  real  physical  system,  rather  than  toy  examples  where  the  model  error  is  known  a 
priori.  Of  particular  interest  to  us  are  methods  which  aim  to  evaluate  predictions  of  high  dimensional 
systems  like  the  atmosphere,  methods  to  best  combine  different  models,  and  determination  of  aims 
which  are  both  achievable  by  the  forecaster  and  useful  for  the  user. 

OBJECTIVES 

We  wish  to  establish  exactly  how  it  is  that  uncertainty  in  the  initial  condition  and  imperfections  in  the 
model  limit  the  prediction  of  nonlinear  dynamical  systems  (NDS)  like  the  atmosphere  and  ocean.  Of 
particular  interest  is  the  evaluation  and  interpretation  of  ensemble  forecasts,  now  routinely  made  at  the 
European  Centre  for  Medium-Range  Weather  Forecasting  (ECMWF)  and  the  National  Center  for 
Environmental  Prediction  (NCEP).  Particular  questions  include  (i)  how  to  best  distribute 
computational  resources  between  using  larger  ensembles  versus  higher  resolution  models,  (ii)  when  to 
select  a  best  model  versus  when  to  use  an  ensemble  over  models,  (iii)  how  to  evaluate  current 
ensemble  formation  schemes  and  (iv)  whether  the  answers  to  these  questions  change  when  regional 
forecasts  are  of  interest.  While  the  focus  is  on  geophysical  fluid  dynamics,  the  insights  gained  are 
applicable  across  a  wide  range  of  forecasting  and  control  systems,  including  mechanical  and  industrial 
processes,  and  general  questions  of  data  analysis  and  assimilation. 

APPROACH 

Our  initial  approach  is  to  develop  tests  of  internal  consistency  and  then  apply  them  in  the  context  of 
simple  numerical  and  laboratory  systems  where  more  reliable  statistics  can  be  gathered  than  in  the 
operational  systems.  These  initial  applications  often  suggest  improvements  in  the  technique  and 
understanding  which  ease  application  to  operational  models,  where  my  group's  approach  has  already 
uncovered  several  fundamental  inconsistencies,  both  in  model  formulation  and  evaluation. 
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Leonard  Smith  (Senior  Research  Fellow,  Pembroke  College,  Oxford;  Reader,  London  School  of 
Economics) 

Mark  Roulston  (Junior  Research  Fellow,  Pembroke  College,  Oxford)  is  investigating  methods  for 
generating  probabilistic  forecasts  using  multiple  ensembles.  He  is  also  studying  system- model  pairs  for 
examining  imperfect  model  issues  and  the  general  questions  of  the  selection  of  model  structure.  An 
example  of  the  latter  is  the  choice  between  linear  and  nonlinear  modeling  strategies,  as  exemplified  by 
his  work  on  modeling  the  dynamics  of  El  Nino-Southem  Oscillation. 

Alexandra  Guerrero  (D.Phil  student)  has  just  joined  the  group  and  is  investigating  the  use  of  ensembles 
consisting  of  multiple  models.  Her  background  is  in  statistics  and  she  is  exploring  the  data  based 
analysis  and  identification  of  stochastic  systems  including  issues  of  model  selection  in  a  nonlinear 
stochastic  framework. 

WORK  COMPLETED 

A  general  overview  of  ensemble  forecasting  and  the  limitation  of 

Current  ensemble  formation  schemes,  focusing  on  the  perfect  model  scenario,  was  completed  (Smith, 
Ziehmann  and  Fraedrich  1999).  Among  the  conclusions  presented  were  that  none  of  the  popular 
analytic  methods  of  estimating  predictability  based  upon  linearization  of  the  system  equations 
(Lyapunov  exponents,  doubling  times,  etc.)  gave  real  insight  into  local  predictability:  fully  nonlinear 
ensemble  forecasts  are  required. 

Tests  we  developed  were  used  to  evaluate  the  duration  of  the  linear  regime  in  operational  weather 
forecasts,  this  is  a  crucial  parameter  in  determining  the  consistency  of  ensemble  formation  techniques 
based  on  singular  vectors;  this  work  is  now  in  review  (Gilmour,  Smith  and  Buizza  2000).  It  indicates 
that  the  duration  of  the  linear  regime  in  operational  weather  forecasting  models  is  much  shorter  than 
commonly  assumed,  implying  that  some  operational  ensemble  formation  schemes  are  not  internally 
consistent  (Gilmour  1999). 

A  new  variety  of  shadowing  trajectory  was  defined  (see  Smith  2000a),  and  used  to  show  the 
limitations  model  error  puts  on  data  assimilation  techniques.  In  particular,  the  general  conclusions  of 
Lorenz  and  Emmanuel  (1988)  on  the  relative  merits  of  different  schemes  for  making  adaptive 
observations  (selecting  which  additional  observations  to  make  among  a  wide  range  of  potential 
observations),  were  significantly  modified,  and  extended  to  the  imperfect  model  case.  Methods  to  test 
the  relevance  of  our  conclusions  in  an  operational  setting  were  developed  (Hansen  and  Smith  2000). 

A  new  cost  function  was  derived  showing  the  limitations  of  the  common  one  step  least  squared  error 
criterion;  at  present  applications  are  limited  to  low  dimensional  systems,  but  both  deterministic  and 
stochastic  models  are  considered  (McSharry  and  Smith  1999). 

Review  of  the  goals  of  ensemble  forecasting  in  the  face  of  real  world  limitations  was  begun,  and 
alternative  (achievable)  targets  were  identified  to  replace  ideal  (but  unphysical)  targets.  The  existence 
of  an  infinite  number  of  'indistinguishable  states'  has  been  proven,  even  give  a  perfect  model  and 
observations  extending  back  arbitrarily  far  into  the  past.  This  makes  the  formation  of  a  perfect 
ensemble  unattainable  even  given  a  perfect  model  and  calls  the  possibility  of  making  accurate  forecasts 
of  probability  density  functions  (PDFs)  into  question  (Smith  2000a,  Judd  and  Smith  2000). 


Quantification  of  the  relative  benefits  of  ensemble  size  versus  model  accuracy  have  been  derived  in  the 
special  case  of  Gaussian  uncertainties;  the  smallest  ensemble  size  likely  to  capture  "truth"  within  the 
bounding  box  of  the  ensemble  (an  alternative  to  aiming  for  an  accurate  PDF),  has  been  computed  as  a 
function  of  the  model  bias  and  the  standard  deviation  of  the  perturbations  used  to  form  the  ensemble. 

Using  models  suggested  by  Lorenz  (1996),  a  scheme  has  been  developed  to  provide  a  test  bed  for 
exploring  the  improvement  of  parameterization  of  sub-gridscale  processes  and  the  understanding  of 
model  sensitivity  to  parameters. 

Minimal  Spanning  Trees  (MST)  have  been  used  to  generalize  the  Talagrand  diagram  to  higher 
dimensions,  thereby  allowing  more  robust  assessment  of  ensemble  forecasts  (Smith  2000a). 

RESULTS 

A  skill  score,  specifically  designed  for  probabilistic  forecasts  has  been  proposed  by  Roulston  and  is 
now  being  investigated.  It  has  now  been  shown  that  this  skill  score  is  a  more  relevant  measure  of 
forecast  skill  than  previous  skill  scores  (e.g.  Brier  Score)  in  cases  where  the  users  of  the  forecast  must 
essentially  "gamble"  on  the  weather. 

A  rational  method  for  combining  ensemble  forecasts  with  knowledge  of  climatology  has  been 
developed.  It  has  been  found  that  this  method  leads  to  a  reduction  in  the  variability  of  the  skill  of 
probabilistic  forecasts  in  the  context  of  perfect  ensembles.  The  utility  of  this  method  will  be  tested 
using  ECMWF  ensemble  forecasts. 

IMPACT/APPLICATIONS 

Current  results  indicate  that  even  in  paradigm  chaotic  systems  like  nonlinear  electric  circuits,  chaos 
provides  less  of  a  barrier  to  prediction  than  model  error  (Smith  2000a, 2000b).  We  believe  that  this  may 
be  true  for  most  physical;  in  particular  we  have  found  that  persistent,  state  dependent,  error  exists  in 
the  ECMWF  operational  model  (Orrell  el  al.  2000).  This  work  is  the  focus  of  the  doctoral  thesis  of  Mr. 
David  Orrell,  a  D.Phil  student  in  our  group  funded  by  ECMWF  and  the  British  government.  A  large 
impact  should  be  expected  wherever  this  is  found  to  be  the  case,  as  it  suggests  a  redistribution  of 
resources  between  model  development,  data  assimilation  and  ensemble  forecasts. 

The  development  of  standard  system-model  pairs  in  which  the  model  is  an  imperfect  representation  of 
the  system.  Such  pairs  should  enable  results  valid  in  the  perfect  model  context  to  be  generalized  to 
imperfect  models. 

Further  work  on  generating  probabilistic  forecasts  using  multiple  ensembles  and  multiple  models 
should  lead  to  greater  user  value  for  operational  weather  forecasts.  In  particular,  we  are  translating 
ensemble  weather  forecasts  into  ensemble  forecasts  of  quantities  of  direct  interest  to  the  user  (e.g.  the 
distribution  of  likely  electricity  demand  or  the  distribution  of  the  likely  landing  rate  (planes  per 
minute)  on  a  given  runway.) 

We  have  clarified  a  technical  confusion  in  the  atmospheric  science  literature:  two  distinct  types  of 
localized  Lyapunov  exponents  with  rather  different  properties  were  both  being  called  "local  Lyapunov 
exponents"  leading  to  incorrect  conclusions.  These  have  been  clarified  (Smith,  Ziehmann  and  Kurths 
2000). 


TRANSITIONS 


Prof.  David  Kenning  (Engineering  Dept.,  Oxford)  is  using  the  modeling  techniques  we  have  developed 

to  study  the  dynamics  of  boiling  and  heat  transfer.  Dr.  Jost  von  Hardenberg  has  joined  our  group  to 

study  spatial-temporal  patterns  in  boiling,  and  brings  expertise  both  in  that  field  and  in  geophysical 

fluid  dynamics  in  general. 

RELATED  PROJECTS 

1-  Vortex  system-model  pair.  A  system-model  pair  is  being  developed  by  Mr.  Edgar  Perez,  a  D.Phil 
student  funded  by  the  government  of  Mexico.  The  system  consists  of  a  collection  of  n  point 
vortices  which  is  completely  defined  by  2n  first  order  ordinary  differential  equations.  The  flow 
field  associated  with  this  system  is  modeled  using  a  finite  element  or  spectral  code.  This  system- 
model  pair  will  be  used  to  investigate  issues  that  arise  when  model  are  imperfect,  in  particular  that 
even  a  low  order  system  may  require  a  high  resolution  model. 

2-  Numerical  simulation  of  physical  systems  and  the  role  of  error  in  the  model-class.  Mr.  Reason 
Machete  has  completed  a  MSc  project  working  with  both  Smith  and  Roulston:  the  project 
contrasts  the  integration  of  the  Moore-Spiegel  system  using  numerical  methods  versus  the  output 
of  a  custom  built  analogue  electrical  circuit.  It  is  hoped  that  this  can  provide  another  system- 
model  pair  for  investigating  imperfect  model  issues. 

3-  The  search  for  response  manifolds  in  the  parameter  space  of  climate  models.  Barbara  Casati  has 
completed  an  Msc  project  (under  the  supervision  of  Smith)  which  investigated  a  method  for 
determining  whether  the  response  of  a  model  to  parameter  changes  can  be  explained  using  a 
reduced  set  of  parameters.  No  evidence  for  the  existence  of  such  manifolds  was  found. 

4-  Information  theoretic  based  decompositions.  British  Energy  (formerly  Nuclear  Electric)  has 
funded  a  D.Phil  studentship  which  Mr.  Liam  Clarke  has  taken  up  under  my  supervision.  The  aims 
of  this  project  include  developing  nonlinear  extenstions  of  Proper  Orthogonal  Decomposition 
(POD)  and  methods  of  model  formation  when  many  multiple  inputs  are  available. 

5-  NCAR  climate  model  output  analysis.  100,000  days  of  perpetual  January  from  NCAR's 
atmospheric  model  is  being  analyzed  using  methods  devised  for  studying  nonlinear  systems;  in 
particular  to  determine  the  time  scales  for  useful  analogies  and  as  an  example  of  how  to  make  the 
choice  between  linear  models  or  nonlinear  models  and  how  it  varies  as  the  duration  of 
observations  increases. 

6-  The  value  of  ensemble  weather  forecasts.  The  value  of  ECMWF  ensemble  forecasts  to  decision 
makers  is  being  determined  in  a  project  funded  by  ECMWF.  The  aim  is  to  establish  the  conditions 
under  which  a  probabilistic  ensemble  forecast  has  value  above  a  traditional  deterministic  forecast. 
The  value  obtained  versus  the  size  of  the  ensemble  will  also  be  found. 

7-  Casino-21.  Our  group  is  active  in  the  formulation  of  an  ensemble  climate  modeling  project  which 
aims  to  use  privately  owned  personal  computers  in  a  massive  parallel  computing  scheme.  One  of 
us  (Smith)  is  on  the  Casino-21  steering  committee,  while  both  Smith  and  Roulston  are  involved  in 
issues  of  experimental  design. 


8-  Linear  and  nonlinear  models  of  El  Nino-Southern  Oscillation  (ENSO).  Roulston  is  leading  a  study 
to  contrast  nonlinear  models  of  ENSO  (forecasts  on  time  scales  from  3  months  to  over  a  year).  A 
general  methodology  is  applied  and  the  relative  quality  of  global  linear  models  (given  the  current 
duration  of  the  data)  are  established. 
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